Skip to main content

Reply to this post | Go Back
View Post [edit]

Poster: Freestone Slice Date: Feb 15, 2014 1:18am
Forum: forums Subject: Why does the WayBack Machine not crawl LiveJournal?

There are actually a couple of sites like this, but LJ is the most recent one I've noticed, so I'm using it as an example. It does have a robots.txt file, but the file doesn't exclude archive.org either specifically or as a blanket restriction. What's up?